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(57) Abstract 

For in a medical environment interac- 
tively constructing relational geometric objects 
that are each defined in an object category 
within a finite set of categories, each object 
gets assigned a number of parameters that col- 
lectively fix the object in question. In partic- 
ular, each object gets assigned one or more 
directional linkings to other such objects in a 
predefined linking structure. Furthermore, var- 
ious such objects are geometrically associated 
with image positions in a medical image that 
has a first dimension and a second dimension, 
whilst adjusting parameter values of various 
such objects to their associated image positions, 
and updating parameters of other objects linked 
therewith according to the structure, for after 
terminating the updating accommodating geo- 
metrical information with respect to the medi- 
cal image. 




j/ 



Une 



view 
tkcimage 








Une 





44 



between 


— 46 




between 


-0.50 




48 — 


1.50 





60 



view 
■ tkcimage 1 




between 


1 1 
56 






point 2d 
480,1112 




glxprop 
colorgreen 


i 

68 




I 

70 



view 
"tkcimage" 


- 




point 2d 
(294,1144) 




I 



72 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Armenia 


FI 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


sz 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


Burkina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BY 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


uz 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Viet Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


Norway 


ZW 


Zimbabwe 


CI 


C6te dTvoire 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


PT 


Portugal 






cu 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






cz 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






DE 


Germany 


LI 


Liechtenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SG 


Singapore 







WO 00/63844 j PCT/EP00/03075 

A method and apparatus for interactively constructing relational geometric objects. 



BACKGROUND OF THE INVENTION 

The invention relates to a system as recited in the preamble of Claim 1. Modern 
medicine is ever more frequently depending on the expert handling of medical images that 
may be produced with a variety of technologies, such as single-shot and multiple shot X-Ray, 
5 computer tomography, magnetic resonance, ultrasound, and others. Subsequent medical 

procedures that are based on those images need precise and detailed descriptions of the various 
objects represented in the image or images, for deciding, administering, and sizing a particular 
treatment. The inventors of the present invention have recognized that interrelating such 
objects in a systematically organized structure would allow better ease-of-handling and 
1 0 improved reliability. This structure would to some extent mirror the relationships of the 
imaged elements in vivo. 

SUMMARY TO THE INVENTION 

In consequence, amongst other things, it is an object of the present invention to 
1 5 provide for a structured handling of various geometric objects in the image or images, in such 
manner that a certain geometrical consistency is maintained, and which handling may be 
effected by a user person, or by a protocol or wizard. Now therefore, according to one of its 
aspects the invention is characterized according to the characterizing part of Claim 1 . 

The invention also relates to a device arranged for practicing the method. 
20 Further advantageous aspects of the invention are recited in dependent Claims. 

BRIEF DESCRIPTION OF THE DRAWING 

These and further aspects and advantages of the invention will be discussed 
more in detail hereinafter with reference to the disclosure of preferred embodiments, and in 
25 particular with reference to the appended Figures that show: 

Figure 1 , a device for practicing the invention; 

Figure 2, hip prosthesis planning; 

Figure 3, ditto after selecting various objects; 

Figure 4, indication of vertebral landmarks; 
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Figure 5, length and angle measurements from calculated vertebral corner 

points; 

Figure 6, an exemplary object structure; 

Figures 7a-7c, the interrelating of image elements in a temporal structure. 

5 

BACKGROUND DISCUSSION OF HANDLING MEDICAL IMAGES 

In an image, geometrical objects will often relate to other geometrical objects. 
Conventionally, this relationship is not enforced by the graphical representation itself, and a 
change made to one object would therefore often necessitate many changes in other objects, 

10 often to the extent to having to redraw or regenerate the complete graphical construction. The 
present invention proposes a method to force the relations between the graphical object 
representations in an electronic and conceptually straightforward manner. 

The objects may have two, three, or even more dimensions, where also time 
could represent a variable. The complete set of objects represents a toolkit, including functions 

15 for measurement, analysis, and construction operations. The relations between objects may be 
purely geometrical, such as that they would be parallel, perpendicular, or concentric with 
regard to each other. Alternatively, such relations may follow a more complex formalism, such 
as one that gives rise to fixing or optimizing a distance, as depending on a set of other objects. 
Still further, the relations between objects may be determined by information in the underlying 

20 image data or in statistical models, such as the local image intensity center of gravity, image 
gradient edges, corners, or other features, statistical shape models, etcetera. The designing of a 
structure of objects may follow from easy-to-learn user interactions. Alternatively, the 
protocol may be preprogrammed, such as for executing complex and/or repetitive tasks. 

The toolkit contains various tool types, that may be elementary or compound in 

25 nature. In the latter case they may be derived from a set of various objects provided with 
primitive types and/or other derivative types. Each object type has a geometrical 
representation that may depend on the image type on which the object is superimposed, of on a 
user's viewpoint, such as the projection direction of a 3D object's view. 

The intuitive and interactive user interface allows the creating of new objects, 

30 the correcting of object positions, the creating of derivative objects in relation to other objects, 
and the amending of remaining undetermined object parameters. Depending on the number 
and types of selected objects, the user sees what new object may be created. Such object then 
will have a predetermined relation with the selected objects. The remaining degrees of 
freedom pertaining to this object may then be interactively updated. 
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Implementation may be straightforwardly done in an object oriented design or 
programming environment. A tool translates into a class, objects translate into instances from 
that class. The relations between objects are maintained in a graph structure. 

By way of example, two fields of application are mentioned. First, the planning 
5 of implants on X-ray images, such as for hip and knee prostheses. Next to a number of 
elementary geometrical objects such as points, lines, and ellipses, a model of a prosthesis 
template is displayed and coupled to the other objects. Note that the prosthesis generally 
comes in a range of different sizes. Second, the planning of the saw planes in a knee 
osteotomy may be done as follows. First, the support structure for the saw is positioned 
10 through several adjusting screws. Next, the remaining degree of freedom for the sawing 

direction is determined. This gives the first sawing plane. Starting from this first sawing plane, 
the sawing is continued through using various sawing guides. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

1 5 Figure 1 shows a device for practicing the invention. Herein, item 20 is a 

medical object, such as a part of the human body. Item 22 is a medical imaging system, such 
as an X-ray device that provides a pattern of relative transmittivity. For other imaging 
technologies such as Magnetic Resonance, the physical background that would produce the 
image intensity is of course different. Item 24 is an image processing device, that may execute 

20 various types of image enhancement other processing. Next, the image is stored in data 

processing device 26, and displayed on user display 28. Through an appropriate user interface, 
such as exemplified through keyboard 32, and generally enhanced by mouse-type or other 
features not shown for simplicity, a user person may select various image positions for 
associating thereto a geometrical object. This selecting will be retrocoupled to device 26 for 

25 subsequent display on item 28. Furthermore, the device 26 may execute various updates of the 
relations if necessary. The overall geometrical structure may be stored in memory 30 that 
accommodates the storage of such data and programs as necessary. If applicable, a structure 
may be stored in memory for repetitive later use. 

By way of example, the following objects and relations may apply: 

30 Point 
Line 

Ellipse (special case: circle) 
Rectangle (special case: square) 
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GeoPath (that is an open/closed sequence of lines or PolyLine, or a sequence of 
Bezier curves or PolyBezier) 

Text (length measurement, angle measurement, area measurement, etc.) 

5 Examples of Relations : 
Point(s) 

one point defines a line, with three parameters: direction, start, stop 
one point defines an ellipse, with three parameters: two radii, and an orientation 
two points define a line, with two parameters: start, stop 
1 0 three points define a circle 

more than two points: line with least squares fitting 
more than three points: circle/ellipse fitting 

Ellipse(s) 

1 5 an ellipse has a center point 

an ellipse has a point on its contour (parameter: orientation) 

an ellipse has a tangent line on a point of its contour (parameter: orientation) 

an ellipse defines another ellipse with the same midpoint (three parameters: orientation and 

radii); in a special case an ellipse defines two circles with the same midpoint and with 

20 short/long axis as respective radii 

two ellipses define a line through both midpoints 

two ellipses define two lines tangent to both ellipses 

an ellipse has a text object that represents its shortest radius 

an ellipse has a text object that represents its longest radius 

25 

Line 

a line defines a line parallel to it (parameter: distance between both lines) 
a line defines a midpoint, or a rational point, i.e. a point at a fixed ratio with respect to the two 
generating points of the line 
30 two lines define an intersection 



Points-Ellipse 

the line through the point and through the midpoint of the ellipse 
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One point and one ellipse: the intersection point of the line through the point and the midpoint 
of the ellipse with the ellipse 

one point and one ellipse: one or two lines through the point, tangent to the ellipse 



10 



Point-Line 

a line parallel to the Line through the Point 

the projected point of Point on the Line 

a line orthogonal to the Line through the Point 

a circle with the Point as midpoint, tangent to the Line 



GeoPath 
a joint on the geoPath 
a tangent line in a point of the geoPath 
a normal line to a point of the geoPath 
1 5 a geoPath branching off in a point of this geoPath 



In particular, the invention may be practiced on three respective different levels 
of interaction. First, a user person may produce a structure completely by hand, such as for 
developing a solution for a new or uncommon situation. Second, the structure can be 

20 protocolized on various levels, so that the system will guide the user how to build the 

structure. Finally, the construction may be fixed and recovered from memory as programmed, 
or after storage of an initial construction. Then it has only to be adapted to the actual instance. 

As a few examples, the following is recited. If the object in question is the 
midpoint of a line drawn between two other points = objects, the shifting of these points will 

25 immediately cause an appropriate shifting of the midpoint. Inversely, the shifting of the 

midpoint may cause the shifting of the line, such as by keeping its length and direction while 
changing its location. Note that the structure of the model will govern and quantify the update. 
For example, the line may be finite or infinite, and an intersection with the infinite line beyond 
the finite line endpoints may be considered valid or not. In the latter case, the action that 

30 moved the intersection beyond the finite line endpoints may be judged invalid, and therefore 
ineffective, so that the action may be undone. 

Figure 2 shows hip prothesis planning, in particular, a part of the various 
geometrical constructions that are needed for planning hip protheses. Various primitives have 
been plotted: two base points of the pelvis, the ellipse approximating the right femur head, and 
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four points on the left femur bone contour. From these primitives several depending objects 
are constructed: a line through the two pelvis base points, the center point of the right femur 
head ellipse, a line parallel to the pelvis line through the right femur head center, the left femur 
head ellipse with its center on the parallel line, the midpoints of both femur contour point 
5 pairs, and the template of a prosthesis on this line through those midpoints. 

Figure 3 shows the same after selecting various geometrical objects. In practical 
usage, this image is colored, and it illustrates that when an object is selected this will be 
highlighted, and by way of example thereby gets a yellow color, the objects that directly 
influence this object, the so-called parent objects, also change their color, such as to red. 

10 Alternatively or additionally, also objects that directly depend on the selected object may have 
their color changed as well, such as to green. Other highlighting techniques would be self- 
apparent to a skilled art person. 

Figure 4 shows indicating of vertebral landmarks, and in particular, a 
construction for obtaining the four corner points of a vertebral body. Two primitive points are 

15 indicated on each of the four vertebral body sides: two endplates and two sides. The point 

pairs are interconnected through lines and the four crossings of successive lines are calculated. 
The crossing point on the lower right is selected (large "y e U° w " diamond): the lines from 
which this is the crossing point (the corresponding parents) are therefore highlighted as well. 

Figure 5, shows further to Figure 4 length and angle measurements from 

20 calculated vertebral corner points: the length of the upper vertebral end-plate and the angle 
between the upper and lower endplate. 

Figure 6 gives an exemplary object structure. This example can be best read 
from bottom to top. First, four 2D points are created in blocks 62, 66, 68, 72, and overlayed in 
the color green, not shown here, on an image that is called tkclmage, symbolized by blocks 50, 

25 54, 56, 60. The coloring itself is symbolized by blocks 64, 70. Further, the midpoints of the 
lines connecting point pairs are calculated in blocks 52, 58, and give rise to a line that is 
represented by block 40. Furthermore, two other points are calculated on the line connecting 
the midpoints with "between -0.5", represented by block 46 and with "between 1.5", 
represented by block 48. The relevant values are relative coordinates with respect to the two 

30 midpoints. In their turn, the latter two points constitute a further line represented by block 44, 
that is displayed on the image as represented by block 42. In practice, such a graph will 
continue with further relations and/or objects not shown for clarity. 

In Figure 6, all relations have been shown as being one-directional. As recited 
supra, all or part of them may be defined as bidirectional. In updating the overall set of 
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objects, often a single run may suffice, in particular, when all relations are one-directional 
only, such as the bottom up run when in Figure 6 one of the four basic points 62, 66, 68, 72 is 
changed. If bidirectional relationships are present as well, it may be that a bottom-up run is 
being followed by a top-down run, and then a second bottom-up run. Other patterns with 
5 more, or with fewer runs, may be suitable for the particular situation or configuration. Of 
course, no more will be executed than really necessary. 

Regarding to the 2D hip prosthesis planning, the following calibration 
procedure is described. The planning in fact depends on an accurate determination of the scale 
factor of the projected objects in the displayed scene from pixels to millimeters. Usually, such 

10 a calibration procedure consists of indicating an object with a known size: a line is drawn on 
this object to present to the system the real length of this line. This will fix the calibration 
factor for subsequent use for all length measurements. This calibration procedure may be 
incorporated into the connected graph, so that all length measurements are linked to this 
calibration factor. The same goes for the size of the implant template which is defined in 

15 absolute measures such as millimeters. Changing the calibration at any time will immediately 
show the effect on all length measurements and on the size of the template. This allows to 
assess the sensitivity of the outcome on the precision of the calibration. 

The above described procedure for hip implant planning is being based on 2D 
projection images. An extension thereof to 3D images, such as are produced by Computed 

20 Tomography is straightforward. Therein, the femur bone may be modeled as a cylinder, the 
femur head as a sphere or as an ellipsoid, the pelvic base as a plane, etcetera. The axis of the 
femur may be approximated by a line through the midpoints of two circles, instead of by a line 
through the midpoints of two lines. Other 2D relations may be generalized directly, such as by 
a line that runs parallel to a plane and through the midpoint of a sphere or of an ellipsoid. 

25 An additional strong advantage of the graph of relational geometric objects is 

that it allows to study the effect of a change in one of the parameters of one of it's objects 
directly. Moving one of the objects on the display screen, or changing the setting of a user 
interface slider that is coupled to an object parameter, will show the effect on all other 
involved objects in a visual manner. Also quantitative analysis of error propagation can be 

30 easily implemented, by adding an error range to each parameter and calculating the effects 
thereof on all related objects. 

Regarding Figures 4 and 5, the previous construction illustrated the indicating 
of landmarks on a single vertebral body. In principle, this procedure must be repeated on all 
relevant vertebral bodies, up to 12 thoracal and 5 lumbar vertebrae. Further, the entire set of 
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landmarks from all vertebral bodies may also be created and maintained with the toolkit 
concept, recited supra. This allows the calculating of additional quantitative parameters over 
the entire set of objects, such as the overall length of the spinal column, the maximum 
curvature, or the largest offset of a particular vertebral midpoint with respect to a vertical axis. 
5 Since this procedure is time-consuming, some automation is desirable, such as 

has been disclosed in Patent Application (PHN 17133) assigned to the 

present assignee in another context. When a limited number of vertebral landmarks have been 
indicated manually, the remaining landmarks can be efficiently interpolated between the 
already known data. According to the present invention, all interpolated landmarks will be 

1 0 updated immediately each time an operator person will move one of the landmarks to a new or 
improved position, until such operator will judge that an optimum will have been attained. 
This procedure will considerably limit the number of points that need to be indicated 
manually. This continual update procedure can be easily implemented with the geometrical 
relational graph concept. The interpolation procedure may be further improved by making use 

15 of a statistical model of heights, widths, and relative positions of all vertebral bodies. 

Figures 7a-7c show the interrelating of image elements in a temporal structure. 
For simplicity, a temporal structuring of a one-dimensional quantity has been shown, so that a 
2D structure results. In it's most advanced realization, the temporal change of a 3D object will 
produce a 4D structure, such as produced by a sequence of cardiac MR measurements or ultra- 

20 fast CT measurements on the moving heart. Segmentation of the heart wall is accomplished by 
tracing the inner boundary or endocardium and also the outer boundary or epicardium. Now, 
manual contour tracing is extremely cumbersome and error-prone, inasmuch as a single 
session will produce hundreds of contours for a single person measured. 

In line with the foregoing, a certain degree of automation may be introduced by 

25 following the teachings of the invention presented supra with reference to the interpolation of 
vertebral body landmarks. One approach is to use each separate contour as a starting point for 
interpolating and/or extrapolating the contours in all other timeslices and or heartbeat phases. 

A concrete implementation as shown in Figures 7a-7c starts from a set of 
contour points on a regular star-shaped pattern that represents three short-axis slices from 

30 different positions or time instants of the heart. The centers of the star pattern and all radii 

propagate to all other slices and phases in a model-based manner. The entire set of contours is 
updated continuously according to the principle of the present invention, until a user would 
judge that the interpolated outline is sufficiently precise. Primary objects have been shown as 
dark circles, interpolated objects as open circles. 
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1 . A method for in a medical environment interactively constructing relational 
geometric objects that are each defined in an object category within a finite set of categories, 
by assigning to each object a number of parameters that collectively fix the object in question, 

characterized by assigning to each object one or more directional linkings to 
5 other such objects in a predefined linking structure, 

geometrically associating various such objects with image positions in a 
medical image having at least a first dimension and a second dimension, and adjusting 
parameter values of various such objects to their associated image positions, whilst updating 
parameter values of other objects linked therewith according to said structure, 
1 0 for after terminating said updating accommodating geometrical information 

with respect to said medical image. 

2. A method as claimed in Claim 1, for use with a plurality of medical images that 
collectively define at least a third dimension as well, through assigning various such linkings 

1 5 across sequenced images in said plurality. 

3. A method as claimed in Claim 1, wherein selected linkings are one-directional 
and others are bidirectional. 

20 4. A method as claimed in Claim 3, wherein said linkings divide into a first group 

that connect all objects in a directed graph, and a second group that run counter to respective 
linkings in said first group. 

5. A method as claimed in Claim 4, wherein said updating is effected in a limited 
25 number of runs with respect to at least one of said first group and said second group. 

6. A method as claimed in Claim 1, wherein said adjusting is conditional on a 
selecting operation with respect to a predetermined object, and said selecting will highlight the 
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predetermined object in a first manner, and other objects connected therewith through 
selective types of linking in a second, different manner. 

7. A method as claimed in Claim 1 , whilst introducing a calibration object into 
5 said linking structure. 

8. A method as claimed in Claim 1 , wherein a particular sequence of similar 
objects features corresponding sub-objects, and said updating is performed through 
interpolating either with respect to corresponding sub-objects across said sequence, and/or 

10 with respect to various sub-objects pertaining to a single object. 

9. A method as claimed in Claim 1, wherein at least one of said objects has a 
predetermined amount of uncertainty introduced in one of it's parameter values, and said 
uncertainty amount is translated into an uncertainty parameter value of other objects. 

15 

10. A method for in a medical environment interactively handling relational 
geometric objects that are each defined in an object category within a finite set of categories 
and each have assigned a number of parameters that collectively fix the object in question, and 
furthermore one or more directional linkings to other such objects in a predefined linking 

20 structure, 

characterized by geometrically associating various such objects with image 
positions in a medical image having at least a first dimension and a second dimension, and 
adjusting parameter values of various such objects to their associated image positions, whilst 
updating parameter values of other objects linked therewith according to said structure, 
25 for after terminating said updating accommodating geometrical information 

with respect to said medical image. 



A device being arranged for practicing a method as claimed in Claims 1 or 10. 
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• User indicated points. 

o Automatically interpolated points. 
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